Logic Program Induction using MDL and MAP: An Application to Grammars

Probabilistic programs provide an appealing language for describing mental theories, because they are Turing complete: any computable process may be described as a program. Program induction is the problem of inferring theories, in the form of (probabilistic) programs, that describe some set of observations. Minimum Description Length, or MDL, is one common approach to program induction [11]. The MDL approach selects the hypothesis (program) such that the sum of the length of the program, along with the length of the data (observations), when encoded with the help of the program, is minimized. Exactly how the data is encoded depends upon the hypothesis; when MDL is used for program induction, the encoding of the data is typically some sort of certificate proving that the program outputs the observations. Instead of MDL, one may also take a Bayesian approach to program induction. This approach involves placing a prior upon programs and calculating, for a given program, the likelihood of the observations. Typically the prior penalizes longer programs [7, pg 385]. In many situations, such as polynomial curve fitting, the MDL and Bayesian approaches coincide: the MAP hypothesis and the hypothesis with minimal description length are the same [7, pg 392]. In this project, I explore the problem of evaluating candidate programs to explain some number of observations, using both the MDL and MAP criteria. I focus on grammar induction: inferring the process (probabilistic program) that generates phrases found within a corpus. Both MDL and MAP are used